First DeepSeek-OCR working implementation#7
Conversation
bluebread
changed the title
bluebread
changed the title
|
@sfallah Could you please update this branch with the main repository? They've added bicubic interpolation support for cuda/vulkan backends in ggml-org#17022, which we need for SAM. |
|
@bluebread |
…rol & all native resolution modes work
|
@sfallah I've added an argument FYI: DeepSeek-OCR (the original model in PyTorch, not ours) doesn't seem to work with the testing picture in tools/mtmd. I guess it's because DeepSeek-OCR wasn't trained on images like that, probably? Perhaps you could take a look when you have some free time. BTW, should we add DeepSeek-OCR support to llama-server as well? |
|
@bluebread |
|
@sfallah Thanks! I'm not sure if the model was trained for arbitrary image size or not. Another issue is that it would be less intuitive for users to switch from native resolutions to gundam modes or auto mode selection. I'll probably leave this design decision to the maintainers. BTW today I found that img_tool likely adopts a different bicubic interpolation algorithm from GGML/PyTorch ones. This should explain the numerical instability and why it doesn't work well as the original model, e.g. it cannot recognize the title and authors on the cover of DeepSeek-OCR paper (although it can extract the abstract). I'll work on fixing this later. |
|
I don't see any problem with title and authors here?
|
@sfallah You can try replacing the prompt with "Free OCR" and testing other images. It randomly appears/disappears. A little bit puzzling. Hopefully we can wrap up this feature within a few days and then shift our attention to llama-server, which might be more necessary for most users. |
|
@bluebread |
|
@sfallah Yes, this PR is ready now. We can let the maintainers review it first. I was thinking about how to replace the interpolation with the correct approach, but others might have better solutions, so we don't need to figure it all out before the reivew. |
We need to take care of the CI jobs and make sure that all CI actions are successful. |
|
@bluebread |
|
@sfallah Thanks! I agree that we should open another PR for llama-server. |
|
@bluebread |
|
@sfallah Thank you! I've done some cleanup and fixed tensor naming for image newline and view separator. Could you help with: replace with some constants with configurable hyperparameters from metadata and add some comments to explain the DeepSeek-OCR algorithm for better readability. I'm going to fix the LM part. |
) * ggml: backend-agnostic tensor parallelism * support for GPT-OSS, Qwen 3 MoE * partial Vulkan fix * add support for 4/8 GPUs * unconditional peer access * re-use buffers + ggml contexts * fix output pattern * NCCL support * GGML: HIP: add RCCL support * Remove shfl and AllReduce from backend interface * move allocation workaround out of ggml-alloc.c * 2d tensor set/get support * Fix the seg fault without NCCL * Apply suggestion from JohannesGaessler * support for tensor dims % n_devs != 0 * fix view_offs scaling * arbitrary num. of GPUs/tensor split * fix compilation * better granularity estimate * Support device-specific host buffer types if all underlying backends expose the same type. This allows using pinned memory instead of pageable memory for CUDA. Fix compilation errors. * partial Qwen 3 Next support * Fix qwen3 30b (#8) * Fix crash with Qwen-30B-A3B Q4_0 Qwen-30B-A3B Q4_0 has an intermediate dimension of 768. Using a granularity of 256 forces an uneven split between GPUs, which is not supported by the current implementation. * Decide block size based on tensor quantization type * Fix crashes due to KV cache serialization (#9) KV cache serialization requires non-zero offsets on the tensor. Add support in the meta backend to set/get a tensor with a non-zero offset. * metal : fix build (#7) * static memory allocations, fix usage count * fix tensor granularity * more even memory distribution * use BF16 for allreduce * rebase fixup * better error message for unsupported architectures * Fix device mismatch during scatter of allReduce. (#11) There is a mismatch between the dst buffer device and the backend device, causing the use of sync copies * Enable the previous allreduce implementation. It is better in both perf and stability (ggml-org#12) * delay AllReduce for Moe for less I/O * build : clean-up compile warnings * backend : move most of the meta backend API to ggml-backend-impl.h * cont : hide unused public API in the implementation * llama : use llama_device + remove ggml_backend_dev_is_meta() * ggml-backend : remove unused alloc include * minor : remove regex include * ggml : introduce ggml-ext.h for staging new APIs * rebase fixup * fix tests * llama : more robust logic for determining Meta devices (ggml-org#16) * llama : more robust logic for determining Meta devices * cont : fix devs size check Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * cont : fix log type Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * disable roundtrip for meta backend * fix arch selection * Qwen 3.5 support * fix Gemma 4 MoE * fix OpenVino, SYCL * fix test-llama-archs for CPU-only builds * Fix Qwen 3.5 MoE * disable meta backend tests for WebGPU * tests : filter CPU-based devices from the Meta backend tests (ggml-org#17) * meta : formatting, naming, indentation (ggml-org#18) * formatting : llama-model.cpp * formatting : ggml-ext.h * formatting : ggml-backend-meta.cpp * meta : add TODO * add documentation * better error messages * fix GPT-OSS --------- Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* spec: support MTP * fix batch size * rename files * cont : simplify (#7) * MTP: clean-up (#9) * MTP: clean-up * review: use llama_context_type instead of llama_graph_type * review: remove llama_model_has_mtp * review: fix convert issues * convert: fix pycheck * review: formatting * use `mtp-` for identifying mtp models * convert: fix mtp conversion * mtp -> draft-mtp * remove unused llama_arch * add need_embd in speculative * llama: allow partial seq_rm for GDN models for speculative decoding Currently speculative checkpoint needs to restart from a checkpoint after some draft tokens are not accepted, this leads to some wastage in running the target again. This PR adds the ability to rollback upto `draft_max` by storing the GDN intermediates. * fix pending state * vulkan: add GDN partial rollback * meta: extend check to axis 1 * metal: add GDN partial rollback Extend the gated delta net kernel to store intermediate states for partial rollback support on the Metal backend. - Add K (snapshot slot count) as a function constant - Read input state from slot 0 of the 3D state tensor - Write intermediate states to different slots during token loop - For K=1, maintain backward-compatible single-slot behavior Ref: ggml-org@8c05923 Assisted-by: llama.cpp:local pi * delta_net_base: use ggml_pad instead of new_tensor * review: add need_rs_seq * review: rename part_bounded to n_rs * review: deslop comments * review: rename, add asserts * server : adjust checkpoint logic (#11) * server : adjust checkpoint logic * cont : rm asserts * server-context: fix early exit * spec : fix compatibility with n-gram and add TODOs (ggml-org#13) * metal : cleanup * llama : fix faulty bitwise check in recurrent memory * server : disable RS-based MTP in combination with other spec types * spec : add TODOs * cont : fix comment * cont : update comment * common : fix logic for ngram + mtp compat * llama-memory: enable checkpointing with partial rollback * cont: add test-case for loading into a dirty ctx * llama-memory-recurrent: clear rs_idx in clear * download: fix mtp path * llama-arch: fix enorm op * docs: update docs * conversion: fix type annotations --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2 participants
@sfallah Working towards finishing debugging the vision model and a first functionable implementation for DeepSeek-OCR this weekend. We've made a lot of progress!!
Edit: I finished debugging, and everything magically works! Going to debug other modes next. FYI: The interpolation algorithm used in llama.cpp doesn't quite align with PyTorch's version and can produce slightly different results. It can be confusing when you cares about the correctness.
Changes
ggml_flash_attn_extin the SAM encoderget_rel_poswith bilinear interpolation hack (by reshaping tensor to [*, 1, C])clip-impl.he.g. print_tensor_info, print_tensor_sum, save_tensor_to_fileBug Fixes
rel_pos_resizednot being used inget_rel_pos